So it turns out there's actually very little public domain music available -- at least in the USA. Even those 100 year old scritchy-scratchy recordings aren't in the public domain. Here's an explanation why:
http://www.publicdomainsherpa.com/public-domain-sound-recordings.html
The author of this page has an excellent list of public domain recordings here:
http://www.publicdomainsherpa.com/public-domain-recordings.html
You will note a significant number of recordings from:
*** The John and Ruby Lomax 1939 Southern States Recording Trip ***
At the time these were downloaded (April 2014) it was possible to retrieve these MP3s from this fairly low-tech site:
http://memory.loc.gov/ammem/lohtml/lomaxbibAudios1.html
These recordings may have moved as the Library of Congress is upgrading its archives.
From http://en.wikipedia.org/wiki/Ruby_Terrill_Lomax#Field_recordings
The remaining public domain music & spoken word samples you see in this folder were largely retrieved from:
PUBLIC DOMAIN MUSIC:
http://en.wikipedia.org/wiki/Wikipedia:Sound/list
http://www.gutenberg.org/browse/categories/3
http://musopen.org/
http://us.metamath.org/mpegif/mmmusic.html
http://www.archive.org/details/audio
archive.org public domain:
http://memory.loc.gov/ammem/browse/ListSome.php?format=Sound+Recording
PUBLIC DOMAIN SPOKEN WORD:
The George W. Bush Public Domain Audio Archive